{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/uptrain-ai/uptrain/blob/main/examples/open_source_evaluator_tutorial.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 align=\"center\">\n",
    "  <a href=\"https://uptrain.ai\">\n",
    "    <img width=\"300\" src=\"https://user-images.githubusercontent.com/108270398/214240695-4f958b76-c993-4ddd-8de6-8668f4d0da84.png\" alt=\"uptrain\">\n",
    "  </a>\n",
    "</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using UpTrain with the Open-source EvalLLM Evaluator\n",
    "\n",
    "You can evaluate your responses via the open-source version by providing your API keys to run evaluations. UpTrain leverages a pipeline comprising LLM calls for the same. Note that the evaluation pipeline runs on UpTrain's server but none of the data is logged."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For now, we support these models for our open-source version.\n",
    "- GPT model (OPENAI API KEY is needed)\n",
    "- Claude (ANTHROPIC API KEY is needed)\n",
    "- Azure OpenAI Service (AZURE API KEY is needed)\n",
    "- Mistral (MISTRAL API KEY is needed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install UpTrain\n",
    "\n",
    "Run the following commands in your terminal to install UpTrain:\n",
    "```bash\n",
    "pip install uptrain\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from uptrain import EvalLLM, Evals, CritiqueTone, Settings\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Create your data\n",
    "\n",
    "You can define your data as a simple dictionary with the following keys:\n",
    "\n",
    "- `question`: The question you want to ask\n",
    "- `context`: The context relevant to the question\n",
    "- `response`: The response to the question"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data = [{\n",
    "    'question': 'Which is the most popular global sport?',\n",
    "    'context': \"The popularity of sports can be measured in various ways, including TV viewership, social media presence, number of participants, and economic impact. Football is undoubtedly the world's most popular sport with major events like the FIFA World Cup and sports personalities like Ronaldo and Messi, drawing a followership of more than 4 billion people. Cricket is particularly popular in countries like India, Pakistan, Australia, and England. The ICC Cricket World Cup and Indian Premier League (IPL) have substantial viewership. The NBA has made basketball popular worldwide, especially in countries like the USA, Canada, China, and the Philippines. Major tennis tournaments like Wimbledon, the US Open, French Open, and Australian Open have large global audiences. Players like Roger Federer, Serena Williams, and Rafael Nadal have boosted the sport's popularity. Field Hockey is very popular in countries like India, Netherlands, and Australia. It has a considerable following in many parts of the world.\",\n",
    "    'response': 'Football is the most popular sport with around 4 billion followers worldwide'\n",
    "}]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## 1. Evaluations via OPENAI API KEY"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get your OpenAI API key\n",
    "\n",
    "You can get your OpenAI API key [here](https://platform.openai.com/account/api-keys)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "OPENAI_API_KEY = \"sk-*********************\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create an EvalLLM Evaluator\n",
    "\n",
    "Before we can start using UpTrain, we need to create an EvalLLM Evaluator. You can do this by passing your API key to the `EvalLLM` constructor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "settings = Settings(model = 'gpt-3.5-turbo-1106', openai_api_key=OPENAI_API_KEY)\n",
    "eval_llm = EvalLLM(settings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate your data\n",
    "\n",
    "Now that we have our data, we can log it and evaluate it using UpTrain. We use the `evaluate` method to do this. This method takes the following arguments:\n",
    "\n",
    "- `data`: The data you want to log and evaluate\n",
    "- `checks`: The evaluations you want to perform on your data\n",
    "\n",
    "You can find the list of all available evaluations [here](https://docs.uptrain.ai/key-components/evals)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2024-01-30 19:38:17.498\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36muptrain.framework.evalllm\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m104\u001b[0m - \u001b[1mSending evaluation request for rows 0 to <50 to the Uptrain\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "results = eval_llm.evaluate(\n",
    "    data=data,\n",
    "    checks=[Evals.CONTEXT_RELEVANCE, Evals.FACTUAL_ACCURACY, Evals.RESPONSE_RELEVANCE, CritiqueTone(persona=\"teacher\")]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get your results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "   {\n",
      "      \"question\": \"Which is the most popular global sport?\",\n",
      "      \"context\": \"The popularity of sports can be measured in various ways, including TV viewership, social media presence, number of participants, and economic impact. Football is undoubtedly the world's most popular sport with major events like the FIFA World Cup and sports personalities like Ronaldo and Messi, drawing a followership of more than 4 billion people. Cricket is particularly popular in countries like India, Pakistan, Australia, and England. The ICC Cricket World Cup and Indian Premier League (IPL) have substantial viewership. The NBA has made basketball popular worldwide, especially in countries like the USA, Canada, China, and the Philippines. Major tennis tournaments like Wimbledon, the US Open, French Open, and Australian Open have large global audiences. Players like Roger Federer, Serena Williams, and Rafael Nadal have boosted the sport's popularity. Field Hockey is very popular in countries like India, Netherlands, and Australia. It has a considerable following in many parts of the world.\",\n",
      "      \"response\": \"Football is the most popular sport with around 4 billion followers worldwide\",\n",
      "      \"score_context_relevance\": 1.0,\n",
      "      \"explanation_context_relevance\": \"['The extracted context can answer the given user query completely as it provides information about the popularity of different sports globally, including football, cricket, basketball, tennis, and field hockey. It mentions major events, personalities, and the number of followers for each sport, allowing for a comprehensive comparison to determine the most popular global sport. Therefore, the extracted context can answer the user query completely.']\",\n",
      "      \"score_factual_accuracy\": 1.0,\n",
      "      \"explanation_factual_accuracy\": \"1. Football is the most popular sport.\\nReasoning for yes: The context explicitly states that football is undoubtedly the world's most popular sport.\\nReasoning for no: No arguments.\\nJudgement: yes. as the context explicitly supports the fact.\\n\\n2. Football has around 4 billion followers worldwide.\\nReasoning for yes: The context explicitly mentions that football draws a followership of more than 4 billion people.\\nReasoning for no: No arguments.\\nJudgement: yes. as the context explicitly supports the fact.\",\n",
      "      \"score_response_relevance\": 0.5,\n",
      "      \"explanation_response_relevance\": \" \\\"The LLM response contains a little additional irrelevant information because it includes the general statement that football is the most popular sport, which is relevant to the user query. However, the specific number of followers worldwide is not necessary to answer the question and can be considered as a minor addition of irrelevant information.\\\"\\n\\n \\\"The LLM response answers a few aspects by stating that football is the most popular sport with around 4 billion followers worldwide. However, it fails to provide any evidence or data to support this claim. The response also does not address the criteria for determining the most popular global sport, such as viewership, participation, or cultural impact. Without this information, the user will not be highly satisfied with the given answer as it lacks critical aspects of the user query.\\\"\\n\",\n",
      "      \"score_tone\": 0.4,\n",
      "      \"explanation_tone\": \"[Persona]: helpful-chatbot\\n[Response]: The provided response does not align with the specified persona of a helpful chatbot. It lacks any form of assistance or guidance, which is expected from a helpful chatbot. The response simply provides a piece of information without engaging in a conversation or offering any support or assistance to the user.\\n[Score]: 2\"\n",
      "   }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(results, indent=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## 2. Evaluations via ANTHROPIC API KEY"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ANTHROPIC_API_KEY = \"sk-***************************\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from uptrain import Settings\n",
    "settings = Settings(model = 'claude-2.1', anthropic_api_key=ANTHROPIC_API_KEY)\n",
    "eval_llm = EvalLLM(settings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2024-01-30 19:39:49.252\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36muptrain.framework.evalllm\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m104\u001b[0m - \u001b[1mSending evaluation request for rows 0 to <50 to the Uptrain\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "results = eval_llm.evaluate(\n",
    "    data=data,\n",
    "    checks=[Evals.FACTUAL_ACCURACY, Evals.RESPONSE_RELEVANCE, CritiqueTone(persona=\"teacher\")]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "   {\n",
      "      \"question\": \"Which is the most popular global sport?\",\n",
      "      \"context\": \"The popularity of sports can be measured in various ways, including TV viewership, social media presence, number of participants, and economic impact. Football is undoubtedly the world's most popular sport with major events like the FIFA World Cup and sports personalities like Ronaldo and Messi, drawing a followership of more than 4 billion people. Cricket is particularly popular in countries like India, Pakistan, Australia, and England. The ICC Cricket World Cup and Indian Premier League (IPL) have substantial viewership. The NBA has made basketball popular worldwide, especially in countries like the USA, Canada, China, and the Philippines. Major tennis tournaments like Wimbledon, the US Open, French Open, and Australian Open have large global audiences. Players like Roger Federer, Serena Williams, and Rafael Nadal have boosted the sport's popularity. Field Hockey is very popular in countries like India, Netherlands, and Australia. It has a considerable following in many parts of the world.\",\n",
      "      \"response\": \"Football is the most popular sport with around 4 billion followers worldwide\",\n",
      "      \"score_factual_accuracy\": 1.0,\n",
      "      \"explanation_factual_accuracy\": \" Here are the reasoning and judgements for the 5 fact statements:\\n\\n1. Football is the most popular sport.\\nReasoning for yes: The context states that \\\"Football is undoubtedly the world's most popular sport\\\". This directly supports the fact.\\nReasoning for no: No contradicting evidence. \\nJudgement: yes\\n\\n2. Football has around 4 billion followers.\\nReasoning for yes: The context mentions that football draws \\\"a followership of more than 4 billion people\\\". This supports the approximate number stated in the fact.\\nReasoning for no: No contradicting evidence.\\nJudgement: yes\\n\\n3. Football has followers worldwide. \\nReasoning for yes: The context talks about football having major events and personalities that are popular globally. This implies that football has worldwide followers.\\nReasoning for no: No contradicting evidence. \\nJudgement: yes\\n\\n4. Football is a global sport.\\nReasoning for yes: The context states that football has \\\"major events like the FIFA World Cup\\\" which are global events, implying that football itself is a global sport.\\nReasoning for no: No contradicting evidence.\\nJudgement: yes\\n\\n5. Football has the most followers compared to\",\n",
      "      \"score_response_relevance\": 1.0,\n",
      "      \"explanation_response_relevance\": \" \\\"The LLM response directly answers the user's question of 'Which is the most popular global sport' by stating 'Football is the most popular sport'. No additional information is provided beyond identifying football as\\n\",\n",
      "      \"score_tone\": -1.0,\n",
      "      \"explanation_tone\": \" Unfortunately I do not have enough context to fully assess the tone and scoring in the provided example. As an AI assistant without clear guidelines on the persona or expected response, I cannot reliably determine how well the response aligns. I would kindly suggest providing more details on the expected tone, persona traits, and response criteria so that the scoring can be made more objective. Some examples of helpful details would be:\\n\\n- Key traits that define the \\\"helpful-chatbot\\\" persona (e.g. supportive, encouraging, provides resources/options, avoids judgment, etc.)\\n\\n- Whether the response should directly answer the user's question or aim to be more conversational\\n\\n- Any expectations around providing additional info, resources, or suggestions beyond a direct answer\\n\\nWith more context around the expected interaction style and goals, I could then better analyze the response for alignment and provide a more meaningful score. I'm happy to re-assess with any clarifying details you can provide! Please let me know if any part of this is unclear or if you have any other questions.\"\n",
      "   }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(results, indent=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Evaluations via AZURE OPENAI SERVICE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "AZURE_API_KEY = \"************\"\n",
    "AZURE_API_VERSION = \"***********\"\n",
    "AZURE_API_BASE = \"*************\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from uptrain import Settings\n",
    "settings = Settings(model = 'azure/*', azure_api_key=AZURE_API_KEY, azure_api_version=AZURE_API_VERSION, azure_api_base=AZURE_API_BASE)\n",
    "eval_llm = EvalLLM(settings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2024-01-30 19:44:01.308\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36muptrain.framework.evalllm\u001b[0m:\u001b[36mevaluate\u001b[0m:\u001b[36m104\u001b[0m - \u001b[1mSending evaluation request for rows 0 to <50 to the Uptrain\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "results = eval_llm.evaluate(\n",
    "    data=data,\n",
    "    checks=[Evals.CONTEXT_RELEVANCE, Evals.FACTUAL_ACCURACY, Evals.RESPONSE_RELEVANCE, CritiqueTone(persona=\"teacher\")]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "   {\n",
      "      \"question\": \"Which is the most popular global sport?\",\n",
      "      \"context\": \"The popularity of sports can be measured in various ways, including TV viewership, social media presence, number of participants, and economic impact. Football is undoubtedly the world's most popular sport with major events like the FIFA World Cup and sports personalities like Ronaldo and Messi, drawing a followership of more than 4 billion people. Cricket is particularly popular in countries like India, Pakistan, Australia, and England. The ICC Cricket World Cup and Indian Premier League (IPL) have substantial viewership. The NBA has made basketball popular worldwide, especially in countries like the USA, Canada, China, and the Philippines. Major tennis tournaments like Wimbledon, the US Open, French Open, and Australian Open have large global audiences. Players like Roger Federer, Serena Williams, and Rafael Nadal have boosted the sport's popularity. Field Hockey is very popular in countries like India, Netherlands, and Australia. It has a considerable following in many parts of the world.\",\n",
      "      \"response\": \"Football is the most popular sport with around 4 billion followers worldwide\",\n",
      "      \"score_context_relevance\": 1.0,\n",
      "      \"explanation_context_relevance\": \"The extracted context can answer the given user query completely. It states that football is undoubtedly the world's most popular sport with major events like the FIFA World Cup and sports personalities like Ronaldo and Messi drawing a followership of more than 4 billion people. This information clearly establishes football as the most popular global sport. While the context also mentions the popularity of other sports like cricket, basketball, tennis, and field hockey, it emphasizes that football is the most popular sport with the highest number of followers globally. Therefore, the extracted context provides sufficient evidence to answer the user query about the most popular global sport.\",\n",
      "      \"score_factual_accuracy\": 1.0,\n",
      "      \"explanation_factual_accuracy\": \"1. Football is the most popular sport.\\nReasoning for yes: The context explicitly states that football is undoubtedly the world's most popular sport.\\nReasoning for no: No arguments.\\nJudgement: yes. as the context explicitly supports the fact.\\n\\n2. Football has around 4 billion followers worldwide.\\nReasoning for yes: The context explicitly states that football has a followership of more than 4 billion people.\\nReasoning for no: No arguments.\\nJudgement: yes. as the context explicitly supports the fact.\\n\\n\",\n",
      "      \"score_response_relevance\": 0.6666666666666666,\n",
      "      \"explanation_response_relevance\": \"The given LLM response contains a little additional irrelevant information because it mentions that football has around 4 billion followers worldwide. While this information is somewhat related to the user's query about the most popular global sport, it is not necessary to provide an accurate answer. The user is asking for the most popular sport, not the specific number of followers. Therefore, this additional information is only slightly irrelevant and adds some unnecessary detail to the response.\\n \\\"The given LLM response sufficiently answers all the key aspects of the given user query. The LLM response states that football is the most popular sport with around 4 billion followers worldwide. This information directly addresses the user query and provides a clear answer to the question of which sport is the most popular globally. The user will be highly satisfied with this answer as it fulfills their query and provides a logical argument supported by the number of followers.\\\" \\n\",\n",
      "      \"score_tone\": 1.0,\n",
      "      \"explanation_tone\": \"The provided machine-generated response aligns with the specified persona of a helpful chatbot. It provides a factual statement about the popularity of football without any additional tone or emotion. The response is straightforward and informative, which is in line with the persona of a helpful chatbot. \\n\\n[Score]: 5\"\n",
      "   }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(results, indent=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Evaluations via MISTRAL API KEY"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "MISTRAL_API_KEY = \"*********************\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from uptrain import Settings\n",
    "settings = Settings(model = 'mistral/mistral-tiny', mistral_api_key=MISTRAL_API_KEY)\n",
    "eval_llm = EvalLLM(settings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2024-02-23 16:34:13.898\u001b[0m | \u001b[33m\u001b[1mWARNING \u001b[0m | \u001b[36muptrain.operators.language.llm\u001b[0m:\u001b[36mfetch_responses\u001b[0m:\u001b[36m234\u001b[0m - \u001b[33m\u001b[1mDetected a running event loop, scheduling requests in a separate thread.\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1/1 [00:02<00:00,  2.47s/it]\n",
      "\u001b[32m2024-02-23 16:34:16.403\u001b[0m | \u001b[33m\u001b[1mWARNING \u001b[0m | \u001b[36muptrain.operators.language.llm\u001b[0m:\u001b[36mfetch_responses\u001b[0m:\u001b[36m234\u001b[0m - \u001b[33m\u001b[1mDetected a running event loop, scheduling requests in a separate thread.\u001b[0m\n",
      "100%|██████████| 1/1 [00:01<00:00,  1.56s/it]\n",
      "\u001b[32m2024-02-23 16:34:17.971\u001b[0m | \u001b[33m\u001b[1mWARNING \u001b[0m | \u001b[36muptrain.operators.language.llm\u001b[0m:\u001b[36mfetch_responses\u001b[0m:\u001b[36m234\u001b[0m - \u001b[33m\u001b[1mDetected a running event loop, scheduling requests in a separate thread.\u001b[0m\n",
      "100%|██████████| 1/1 [00:00<00:00,  1.57it/s]\n"
     ]
    }
   ],
   "source": [
    "results = eval_llm.evaluate(\n",
    "    data=data,\n",
    "    checks=[Evals.CONTEXT_RELEVANCE, Evals.RESPONSE_RELEVANCE]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "   {\n",
      "      \"question\": \"Which is the most popular global sport?\",\n",
      "      \"context\": \"The popularity of sports can be measured in various ways, including TV viewership, social media presence, number of participants, and economic impact. Football is undoubtedly the world's most popular sport with major events like the FIFA World Cup and sports personalities like Ronaldo and Messi, drawing a followership of more than 4 billion people. Cricket is particularly popular in countries like India, Pakistan, Australia, and England. The ICC Cricket World Cup and Indian Premier League (IPL) have substantial viewership. The NBA has made basketball popular worldwide, especially in countries like the USA, Canada, China, and the Philippines. Major tennis tournaments like Wimbledon, the US Open, French Open, and Australian Open have large global audiences. Players like Roger Federer, Serena Williams, and Rafael Nadal have boosted the sport's popularity. Field Hockey is very popular in countries like India, Netherlands, and Australia. It has a considerable following in many parts of the world.\",\n",
      "      \"response\": \"Football is the most popular sport with around 4 billion followers worldwide\",\n",
      "      \"score_context_relevance\": 1.0,\n",
      "      \"explanation_context_relevance\": \"{\\n\\\"Reasoning\\\": \\\"The given context mentions that football is the world's most popular sport based on its followership of over 4 billion people. The context also mentions other sports like cricket, basketball, tennis, and field hockey, but it does not provide any information that would challenge the statement that football is the most popular sport. Therefore, the extracted context can answer the given query completely.\\\",\\n\\\"Choice\\\": \\\"A\\\"\\n}\",\n",
      "      \"score_response_relevance\": 1.0,\n",
      "      \"explanation_response_relevance\": \"Response Precision: 1.0{\\n    \\\"Reasoning\\\": \\\"The given response provides the accurate answer to the question and does not include any irrelevant information.\\\",\\n    \\\"Choice\\\": \\\"A\\\"\\n}\\nResponse Recall: 1.0{\\n    \\\"Reasoning\\\": \\\"The given response adequately answers the given question by stating that football is the most popular sport with approximately 4 billion followers.\\\",\\n    \\\"Choice\\\": \\\"A\\\"\\n}\"\n",
      "   }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(results, indent=3))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
